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Abstract 


A decade of experience with e-learning material production gives significant trends overview. From 
the perspective of ICT evolution this means very long time. E-learning is always closely coupled to 
ICT but with the distinction - it always lags behind the advancement in ICT. Among many unknown 
issues one thing is certain — e-learning permanently changes. From the past experiences prediction of 
immediate future e-learning is possible but the prediction for longer time is less accurate. In the article 
chronological e-learning achievements and their connections are presented in order to give an overview 
of trends in e-learning in Slovenia. But despite of the detailed past knowledge fundamental prediction 
error of e-learning future developments was made. Todays hot topics are not better e-learning materials, 
not even broad application of interactive boards but e-testing. Possible immediate future of e-learning 
is therefore application of intelligent e-testing systems. Intelligent computer assisted testing (CAT) is not 
the new topic in e-learning. It is object of many researches but it is not suitable for teachers. It is way 
too complicated for teachers to apply it successfully in their work. E-learning development process for 
intelligent computer assisted test production that suits teachers’ competences is prepared and explained 
in the article. 

Key words: distance education, electronic testing, learning management system, knowledge 
assessment. 


Introduction 


E-learning has started in our country at the turn of the century. First e-learning materials 
were produced for study course Multimedia. Beginning with static web pages oriented to 
low bandwidth internet connection process gradually included more and more high quality 
multimedia elements and finishes with learning management system (LMS). But now new 
trends are considered to produce low-computer power and electricity efficient e-learning 
materials suitable for portable devices (BBC, 2012). 

Today almost every teacher knows what multimedia is and how to apply it in the 
classroom. Multimedia has influenced many new e-learning materials development and transfer 
of the knowledge to the primary school. Looking back to the beginning of e-learning several 
time periods can be identified in Slovenian teachers’ education: 


Period | — Initial research (1996 — 1998) 
Period 2 — The beginning (1998 — 2002) 
Period 3 — Maturity (2002 — 2006) 
Period 4 — Restructuring (2006 —2010) 
Period 5 — Mainstream (2010 — today) 
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Trends overview shows students want better e-learning materials but the teachers turn 
into different direction — e-tests. E-testing in education is not new but requires some training. 
Free authoring tools enable simple tests preparation. More elaborate e-testing requires 
different development process which is beyond the competences of Slovenia's teachers. But 
in the article simpler process is going to be explained that have a potential to introduce better 
e-testing to lower levels of education (primary and secondary schools). 


Programmed Learning Paradigm in E-learning Material Development 


During the last decade many e-learning materials development projects were conducted 
in Slovenia. During this time new ideas emerged and old ideas were reinvented in new or more 
aesthetically pleased manner (i.e. Apple iBook). Research has shown that Web 2.0 in “modern 
education” is not mandatory for success of e-learning materials. Some old fashioned e-resources 
are still highly regarded and frequently used by students (Repnik, Kra8na, & Kauéié, 2012). 

Implementation of a digital augmented, blended, assisted, collaborative, etc. learning 
requires a significant teachers’ effort. Teachers should use contemporary approaches and 
technology, and prepare high quality visually appealing e-learning materials. It is expected 
that teachers should produce such learning materials for their needs. This is not true though. 
Teachers are incapable to follow the advancement in technology and its effective use. Computer 
technology in education is therefore mostly used for writing text and web browsing (Gerlié, 
Bratina, Vetek, & Pungartnik, 2011). 

In the last few years LMSs are used in Slovenia’s educational institutions. Workshops 
for their use were organized as in-house training or as LLL (Lifelong Learning) activities. 
Teachers and students are now reasonably familiar with LMSs. But in general LMSs are still 
mostly used for learning material distribution, messaging and data warehouse for students’ 
assignments. An E-learning material produced with programmed learning paradigm receives 
higher students’ appreciations but requires e-testing phase after each topic. In order to better 
utilize LMS’s abilities several demands for additional training from lecturers were expressed 
recently. According to the students’ demands expectation was that teachers would like to 
upgrade their learning materials or use advanced types of communication. Surprisingly most 
desirable demand was how to efficiently use e-testing. 


Computer Based Testing and Computer Adaptive Tests 


Computer based testing (CBT) (Parshall, Spray, Kalohn, & Davey, 2002) and technology- 
based assessment (TBA) can be fast (but processor intensive). Students get feedback as soon as 
they finish the test (i.e. submit all answers to the system). Traditional e-tests enable students to 
answer questions in random order until they submit results or run out of time. 

It is evident that all tests are not equally suitable in every situation. With the increase 
of complexity potential problems increase. Positive or negative question type can be used to 
examine the knowledge. Positive questions are usually easier for examinee than negative. The 
questions therefore are: 

e How to explain the situation where students answer positive question type and not 

negative? 

e What should be graded: the knowledge or understanding of question? 

Traditional e-tests are linear with the fixed or random questions. It is easy for teachers 
to assess student’s knowledge and (transferable) skills and wider competencies in different 
situations. Students are individuals and process information in different ways to learn a given 
domain topic area (Costello & Mundy, 2009) therefore in the process of knowledge evaluation 
teachers apply adapted assessments principle. 
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Adaptability is the ability to “tailor the difficulty level of each question based on 
the correctness of the previously answered question” (Basu, Cheng, Prasad, & Rao, 2007). 
Adaptive techniques were developed for local systems before the advent of WWW (Barra, 
Iannaccone, Palmieri, & Scarano, 2002). First computerized adaptive testing was proposed in 
1980 (Lord, 1980), and lately adaptive technology is more and more used also in CBT. In the 
literature different naming and abbreviations can be found for adaptive CBT, such as computer- 
adaptive assessment (CAA) and computer adaptive testing (CAT). Adaptive e-tests are e-tests 
in which the number and sequence of test items are adjusted during the test in response to the 
performance of the person being tested (Chua Abdullah & Cooley, 2002)(Chen & Wang, 2010). 
Such tests are: 
e Quicker to administer, 
e Reduce costs in administering tests, 
e Can significantly reduce the number of questions (even for 50%) (Cheng, Rodrigez, 
& Basu, 2009), 

e Are student friendly, and 

e Not less precise than non-adaptive tests (Basu, Cheng, Prasad, & Rao, 2007) (Chua 
Abdullah & Cooley, 2002). 

Their main disadvantage is that questions paths are modeled with probabilistic functions 
(Guzman & Conejo, 2005). In addition, they still do not solve concerns as in CBT: cost of 
technology, technology malfunction, administrator training, and student’s computer skill, but 
these concerns are usually easy to deal with. The theories behind adaptive e-tests are Student 
Model (SM) (Chen & Zhang, 2008) of modeling student behavior and characteristics, and 
Item Response Theory (IRT) (Chen & Wang, 2010)(Guzman & Conejo, 2005)(Chen & Zhang, 
2008) which is a robust well-founded psychometric theory in education assessment. The latter 
is responsible for estimating student knowledge, determining the next question that must be 
posed at each moment, and deciding test finalization (Guzman & Conejo, 2005). Some authors 
name combination of SM and IRT as Computer Adaptive Test Theory (CAT theory) (Danieliené 
& TeleSius, 2008). 

From ICT point of view imitation of teachers in oral exams in CAT is done with methods 
of artificial intelligence (AI) and expert educational systems (Gerlié, 2000). Since not a lot of 
systems are developed and generally used, adaptive assessmentis still “an innovative, online form 
of assessment in which items are presented in a sequence that is dependent on the correctness 
of the examinee s response to the preceding item’(Basu, Cheng, Prasad, & Rao, 2007). Some 
products that reader can find in the literature are: system called Test++ that bases on playing 
game methodology (Barra, Iannaccone, Palmieri, & Scarano, 2002), SIETTE for creating tests 
for self-assessment (Guzman & Conejo, 2005), AITS that uses AI techniques (Hatzilygeroudis, 
Koutsojannis, Papavlasopoulos, & J., 2006), system that uses AI agent technology (Song, Chen, 
& Gao, 2011), while some testing system are compared in (Danieliené & TeleSius, 2008). 

From many research results the true potential of CAT is evident. It provides valid 
knowledge assessment and teach student if needed. If adaptive e-test system recognizes some 
misunderstandings, weaknesses or pitfalls in student’s answers then it selects new path of 
questions where student will be challenged to improve or correct his/her answers according to 
learning by assessment paradigm. It usually starts with a question of medium difficulty, while 
the questions path can be determined on-the-fly or can be predetermined (Cheng, Rodrigez, 
& Basu, 2009). Final grading is determined based on the answers in the questions path. CAT 
theory ensures that the obtained student knowledge estimations do not vary in terms of items 
used in the estimation process (Danieliené & TeleSius, 2008). 

In the nineties we have created “Model of application of computer in education” (Gerlié, 
1999) and one of the topics was Expert educational systems (EES). CATs use similar techniques 
as EES but the aim is different. When we observe intelligent computer aided education in 
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details we can see that differences in the tests and education are actually blurred. If we present 
knowledge with the network in the education we want students to visit all paths but in test we 
want students to find individual paths in limited amount of time. 


Designing CATs 


To prepare adaptive test teacher needs to prepare a blueprint of possible questions/ 
answers and state transition diagram. At least two different structures can be used: decision tree 
(Nanéovska Serbec, Zerovnik, & Rugelj, 2011) or state transition diagram. A modified State 
Transition Diagram from UML (Unified Modeling Language) was used to be friendlier for 
general users — teachers (Figure 1). Therefore start sign and stop sign were omitted since reader 
can easily comprehend that. Colored rounded rectangular were used to indicate meta-questions 
and the rounded rectangular for real questions. Rectangular on the line is used for transition’s 
condition. The rectangular with multiple questions represent the questions’ set. Questions from 
the set can be chosen arbitrary and can be of any traditional e-test type. With this approach a 
focus is set on the knowledge evaluation paths. Grading can be omitted when building the test 
structure. If decision tree are used the paths and grading are built at the same time. Such process 
increase complexity and potential errors. 

The best knowledge evaluation would be if students would present entire knowledge 
about the tested topic. Since there is not enough time teachers often compromise and ask only few 
questions and assess knowledge based on these questions. Teachers know these processes very 
well and understand them completely. Some teachers select questions from their inspiration but 
others prepare random questions on pick-up lists and enable students to choose the questions. 
Questions are usually broader and cover one or more topic of entire knowledge of course. If 
student gives suitable answers and show knowledge about that topic he/she receive grade. But 
if answers raise suspicion into student knowledge then additional questions (more elaborate and 
precise) are required to grade student. Every teacher is familiar with this process. Therefore this 
principle is applied in the design process of adaptive computer tests. 


Question topic 


Question 1.1 Question 2.1 Decision n.1 


———————_— 

Question 1.2 [ Question 21° | Question n.1.1 
ee 
os Question n.1.2 


ee 












Route 1 


Decision n 





















Decision n.2 








Question n.2.1* 





Question n.2.2* 











Question n.2.2* 





Eau 


Question n.2.2* 


Figure 1: Proposed schema for CAT design. 
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From the initial point (question topic) multiple routes can be used. Therefore student 
just picks one of them and according to the selected route questions that go with this route are 
evaluated. From the Figure lwe see that student can get mandatory question (1.1 — 1.n) or 
one mandatory question and elective questions (2.1, 2.1*-2.n*) or can go to the lower decision 
level (decision n.1, n.2). This schematic approach is just transformation between what teachers 
ordinarily do and CAT system. Even grading is simple. Each route has maximum numbers of 
points and student’s point are summed and compared to the maximum number of points. 


b all s. 
grade = == 
} Sl 1 qi 
Where: i index on the question. 


n number of question along the selected route. 
s, student achievement on question i. 
q, number of points on question i. 


There are still unresolved ambiguities how to achieve back stepping on the selected route 
when users feel that corrections of previous answers are needed. At least two approaches can 
be used: (1) student cannot go back or (2) if student goes back and make change the system 
forget its subsequent answers. A debate is still open about grading upon students partial path 
completion. 

For easier understanding example “how to upgrade your computer?” was prepared 
(Figure 2). In general there are three ways to improve computer. The option to buy another 
computer was dismissed since it would not provide any valuable results. Students can pick 
three answers in the first question (regarded as paths for later assessments): upgrade RAM, hard 
drive, or processor. If he/she decides to upgrade RAM then his/her decision is tested with the 
questions “How much RAM he/she wants to add”, “* What operation system he/she has”’, “What 
is his/her current RAM configuration and what is his/her RAM speed’. Not all the questions 
have to be asked but cascade question to check student’s understanding are available (e.g. it is 
useless to upgrade RAM over 4 gigabytes if computer has 32 bit operating system). The schema 
shows that the question “How much RAM he/she wants to add” has subsequent question “What 
operation system he/she has’’. If desired questions can be subdivide into more details. 

The second option, upgrading disk, can be fulfilled with two types of disk (magnetic or 
SSD - solid state disk). If student decides to change magnetic disk in his/her computer he/she 
is going to get three questions: what size of disk he/she wants to have, what speed of new disk 
he/she has in mind, and which technology (referred to type PATA and DMA types or SATA 
and subtypes) is on his/her mind. Each question can have right and wrong answers and the 
combination of answers from all three questions can be valid or not. It therefore may happen 
that answers to each question are true but combinations are not. On the other hand if he/she 
decides to use SSD disk only two additional questions will be asked: the size of SSD, and why 
he/she will use this disk. For example, it is known that Windows XP can be squeezed on 4 GB 
disk but Windows 7 requires around 18 GB. We could also add another question in this topic 
about the price. 

The third option to upgrade processor is prepared in similar fashion. 
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How to upgrade your 
computer? 


Upgrade disk 
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Magnetic type SSD type 





Type of computer? 


Power consumption? 


Ram configuration? 
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Ram speed? Technology? 
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Conclusion 


Figure 2: Schema — example — computer upgrade. 


With the proposed design process for smart testing some examples were made. Some 
teachers with high level of enthusiasm have grasped the idea quickly but some dismiss it 
entirely. Especially old teachers would not like to change their proven assessment principles 
(face-to-face assessment). Younger teachers and assistants were much keener to adopt this since 
they discover positive side-effect of this process. Based on such schema they prepare better e- 
learning materials. Only the time will tell if this will become a blueprint for better e-learning and 
e-testing preparation or it is going to be upgraded and incorporate into another better proposal. 


Conclusion 


Research and development is never a straightforward process. Even if e-learning materials 
are still far from being good teachers do not demand their improvements. In house training 
shows that most desirable topics in e-learning are e-tests. Reasons are clear and expected but 
the rush may not provide satisfactory results. E-tests require a mental change from traditional 
tests. Not all teachers have the ability to switch immediately and some training and support will 
be required. Persons responsible for transition from traditional test to e-test must highlight the 
problems of e-tests and prevent teachers to use inappropriate concept from traditional test (fill-in 
blanks) too often. The poor results may increase frustration of students beyond acceptable level. 
Based on gained experiences several years will pass before application of CAT in Slovenia’s 
LMSs. Authoring tool that enables conditional testing are available but they are not cheap (e.g. 
Articulate Quiz Maker). The technology used in these tools is Flash and is not supported on 
low powered portable devices which gain increased users’ popularity. Even graphical tool that 
would support development of conditional routed e-test is not available. Using flow charts 
building blocks would be feasible but teachers are not familiar with them. Conditional diamond 
from flow chart is much spatial demanding and has little usable space for text. The proposed 
scheme is far from perfect but it is a start and PowerPoint can be used for its design. Prediction 
therefore is that in the near future a GUI (graphical user interface) will be developed to support 
CAT in most used LMSs in more natural way. 
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